Virtio-net-controller is a systemd service running on the DPU, with a user interface frontend to communicate with the background service. An SF representor is created for each virtio-net device created on the host. Virtio-net controller only uses an SF number ≥1000. Refer to section "Scalable Functions" for more information.

SF representor name is determined by udev rules. The default name is in the format of . For example: en3f0pf0sf1001.

Each virtio-net PF/VF requires a dedicated SF and it should be reserved from mlxconfig (see section "VirtIO-net PF Device Configuration"). However, since an SF is a shared resource on the system, there may be other application-created SFs as well. In that case, PF_TOTAL_SF must be updated to consider those SFs. Otherwise, virtio-net is not able to create enough configured PF/VF.

Since the controller provides hardware resources and acknowledges (ACKs) the request from the host's virtio driver, it is mandatory to reboot the host OS first and the DPU second. This also applies to reconfiguring a controller from the DPU (e.g., reconfiguring LAG); unloading the virtio-net driver from guest side is recommended.

Controller systemd service is enabled by default and runs automatically if VIRTIO_NET_EMULATION_ENABLE is true from mlxconfig.

To check controller service status, run:

$ systemctl status virtio-net-controller.service

To reload the service, make sure to unload virtio-net/virtio-pcie drivers on host. Then run:

$ systemctl restart virtio-net-controller.service

To monitor log output of the controller service, run:

$ journalctl -u virtio-net-controller -f

The controller service has an optional configuration file which allows users to customize several parameters. The configuration file should be defined on the DPU at the following path /opt/mellanox/mlnx_virtnet/virtnet.conf.

This file is read every time the controller starts. Dynamic change of virtnet.conf is not supported. It is defined as a JSON format configuration file. The currently supported options are:

ib_dev_p0 – RDMA device (e.g., mlx5_0) used to create SF on port 0. This port is the EMU manager when is_lag is 0. Default value is mlx5_0.

ib_dev_p1 – RDMA device (e.g., mlx5_1) used to create SF on port 1. Default value is mlx5_1.

ib_dev_lag – RDMA LAG device (e.g., mlx5_bond_0) used to create SF on LAG. Default value is mlx5_bond_0. This port is EMU manager when is_lag is 1. ib_dev_lag and ib_dev_p0/ib_dev_p1 cannot be configured simultaneously.

ib_dev_for_static_pf – the RDMA device (e.g., mlx5_0) which the static virtio PF is created on

is_lag – specifies whether LAG is used. Note that if LAG is used, make sure to use the correct IB dev for static PF.

static_pf –

mac_base – base MAC address for static PFs. MACs are automatically assigned with the following pattern: pf_mac→pf_0, pf_mac+1→pf_1, etc.

Note that the controller does not validate the MAC address (other than its length). The user must ensure MAC is valid and unique.

features – virtio spec-defined feature bits for static PFs. If unsure, leave features out of the JSON file and a default value is automatically assigned.

vf –

mac_base – base MAC address for static PFs. MACs are automatically assigned with the following pattern: pf_mac→pf_0, pf_mac+1→pf_1, etc.

features – virtio spec-defined feature bits for static VFs. If unsure, leave features out of the JSON file and a default value is automatically assigned.

vfs_per_pf – number of VFs to create on each PF. This is mandatory if mac_base is specified.

This value does not equal VIRTIO_NET_EMULATION_NUM_VF in mlxconfig. vfs_per_pf ≤ VIRTIO_NET_EMULATION_NUM_VF.

qp_num – number of QPs for each VF. If not specified, then the QP number assigned is taken from its parent PF.

recovery – specifies whether recovery is enabled. If unspecified, recovery is enabled by default. To disable it, set recovery to 0.

sf_pool_percent – determines the initial SF pool size as the percentage of PF_TOTAL_SF of mlxconfig. Valid range: [0, 100]. For instance, if the value is 5, it means an SF pool with 5% of PF_TOTAL_SF is created. 0 means no SF pool is reserved beforehand (default).

PF_TOTAL_SF is shared by all applications. User must ensure the percent request is guaranteed or else the controller will not be able to reserve the requested SFs resulting in failure.

sf_pool_force_destroy – specifies whether to destroy the SF pool. When set to 1, the controller destroys the SF pool when stopped/restarted (and the SF pool is recreated if sf_pool_percent is not 0 when starting), otherwise it does not. Default value is 0.

For example, the following definition has all static PFs using mlx5_0 (port 0) as the data path device in a non-lag configuration:

{  "ib_dev_p0": "mlx5_0",  "ib_dev_p1": "mlx5_1",  "ib_dev_for_static_pf": "mlx5_0",  "is_lag": 0,  "recovery": 1,  "sf_pool_percent": 0,  "sf_pool_force_destroy": 0,  "static_pf": {    "mac_base": "11:22:33:44:55:66",    "features": "0x230047082b" },  "vf": {    "mac_base": "CC:48:15:FF:00:00",    "features": "0x230047082b",    "vfs_per_pf": 100,    "qp_num": 4 }}

The following is an example for LAG configuration:

{  "ib_dev_lag": "mlx5_bond_0",  "ib_dev_for_static_pf": "mlx5_bond_0",  "is_lag": 1,  "recovery": 1,  "sf_pool_percent": 0,  "sf_pool_force_destroy": 0}

To communicate with the service, a user frontend program (virtnet) is installed on the DPU. Run the following command to check its usage:

# virtnet -husage: virtnet [-h] [-v] {hotplug,unplug,list,query,modify,log} ... Nvidia virtio-net-controller command line interface v1.0.9 positional arguments: {hotplug,unplug,list,query,modify,log}** Use -h for sub-command usagehotplug hotplug virtnet deviceunplug unplug virtnet devicelistlist all virtnet devicesqueryquery all or individual virtnet device(s)modify modify virtnet devicelog set log level optional arguments: -h, --helpshow this help message and exit -v, --version show program's version number and exit

Note that each positional argument has its own help menu as well. For example:

# virtnet log -husage: virtnet log [-h] -l {info,err,debug}optional arguments: -h, --helpshow this help message and exit -l {info,err,debug}, --level {info,err,debug}log level: info/err/debug

To operate a particular device, either the VUID or device index can be used to locate the device. Both attributes can be fetched from command "virtnet list". For example, to modify the MAC of a specific VF, you may run either of the following commands:

# virtnet modify -p 0 –v 0 device -m 0C:C4:7A:FF:22:98

Or:

# virtnet modify -u device -m 0C:C4:7A:FF:22:98

The following modify options require unbinding the virtio device from virtio-net driver in the guest OS:

MAC

MTU

Features

Msix_num

max_queue_size

For example:

On the guest OS:

$ echo "bdf of virtio-dev" > /sys/bus/pci/drivers/virtio-pci/unbind

On the Arm side:

$ virtnet modify ...

On the guest OS:

$ echo "bdf of virtio-dev" > /sys/bus/pci/drivers/virtio-pci/bind

It is possible to recover the control and data planes if communications are interrupted so the original traffic can resume.

Recovery depends on the JSON files stored in /opt/mellanox/mlnx_virtnet/recovery where there is a file that corresponds to each device (either PF or VF). The following is an example of the data stored in these files:

{ "port_ib_dev": "mlx5_0", "pf_id": 0, "function_type": "pf", "bdf_raw": 26624, "device_type": "hotplug", "mac": "0c:c4:7a:ff:22:93", "pf_num": 0, "sf_num": 2000, "mq": 1}

These files should not be modified under normal circumstances. However, if necessary, advanced users may tune settings to meet their requirements. Users are responsible for the validity of the recovery files and should only perform this when the controller is not running.

Controller recovery is enabled by default and does not need user configuration or intervention unless a system reset is needed or BlueField configuration is changed (i.e., any of the mlxconfig options PCI_SWITCH_EMULATION_NUM_PORT, VIRTIO_NET_EMULATION_NUM_VF, or VIRTIO_NET_EMULATION_NUM_PF). To this end, the files under /opt/mellanox/mlnx_virtnet/recovery must be deleted.

The first time LAG is configured with a controller, recover files must be cleaned up to ensure the controller does not try to recover devices with the previous IB parent device.

Live update minimizes network interface down time by performing online upgrade of the virtio-net controller without necessitating a full restart.

To perform a live update, you must install a newer version of the controller either using the rpm or deb package (depending on the OS distro used). Run:

For Ubuntu/Debian

dpkg --force-all -i virtio-net-controller-x.y.z-1.mlnx.aarch64.deb

For CentOS/RedHat

rpm -Uvh virtio-net-controller-x.y.z-1.mlnx.aarch64.rpm --force

It is recommended to use the following command to verify the versions of the controller currently running and the one just installed:

virtnet version

If the versions that are correct, issue the following command to start the live update process:

virtnet update --startvirtnet update -s

If an error appears regarding the "update" command not being supported, this implies that the controller version you are trying to install is too old. Reinstalling the proper version will resolve this issue.

During the update process, the following command may be used to check the update status:

virtnet update statusvirtnet update -t

During the update, all existing virtnet commands (e.g., list, query, modify) are still supported. VF creation/deletion works as well.

When the update process completes successfully, the command virtnet update status will reflect the status accordingly.

If a device is actively migrating, the existing virtnet commands will appear as "migrating" for that specific device so that user can retry later.